The Mass Univariate Approach and Permutation Statistics

This chapter describes the mass univariate approach to the analysis of ERP data and the related permutation approach. The mass univariate approach is a powerful way of dealing with the problem of multiple comparisons. It was pioneered in neuroimaging research, where the problem of multiple comparisons is so obvious that it is always explicit rather than implicit. In early neuroimaging studies, investigators conducted a t test or ANOVA on each voxel separately, asking whether the activity in each voxel differed significantly across conditions. They then performed a Bonferroni correction for multiple comparisons. For an effect to be significant after a Bonferroni correction, the original p value must be less than the alpha value divided by the number of comparisons. For example, if you are testing the effect of some manipulation in 1000 individual voxels with a typical alpha of .05, a given voxel would need to have an uncorrected p value of .05/1000 (.00005) to be considered significant. To get a significant effect after this draconian correction, the effect must be enormous! This makes it hard to find significant effects even when there are true differences between conditions (i.e., the Type II error rate is very high). Neuroimaging researchers sometimes call this “being Bonferronied to death.” The Bonferroni correction assumes that each of the tests is independent, which is not true of imaging data (or EEG/ERP data) because the values at adjacent variables (i.e., voxels) are high correlated. Moreover, it imposes a very strict criterion for significance that is not always sensible. The result of these factors is very low statistical power.